fix(hermes): settle direct API usage with complete wire receipts

Export image-owned qualification assets from cloud builds and cover Claude/OpenAI image, question and billing journeys.

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-09 10:13:33 -05:00
1 parent 8b5d075f0c
commit e025ff2286
38 files changed
+1347 -120

No files matched your search

+19 -9
View File
@@ -1571,7 +1571,9 @@ candidate image and the matching controller-owned provider pack described in
The explicit-only `hermes-api-connections` suite adds ten one-turn completion
cells: managed OpenRouter, Anthropic, OpenAI, xAI and Google API accounts on
local and Daytona. Select one exact execution ID. The October 7 authenticated
catalog choices are pending inference qualification, and no production default
catalog choices are pending inference qualification; OpenAI's API cell now
uses `gpt-6-luna`, confirmed by authenticated catalog discovery on October 9
and covered by the reviewed direct-API rate card. No production default
changes. Each fixture creates its personal account through the public
Connections API, removes agent credential overrides, and independently checks
the native run's company/task/agent scope, responsible user, selected account,
@@ -1585,7 +1587,11 @@ The OpenRouter cell additionally waits for its public run's reported wire cost
to settle and checks that the company remains active and the agent idle with
no pause reason before cleanup. Its separate settlement snapshot rejects missing
price evidence, estimates and an unpriced budget pause even when the answer is
correct. Other providers retain completion-only coverage.
correct. Claude and OpenAI API cells require a complete selected-account wire
token receipt, the correct model's explicitly estimated rate-card cost and
healthy post-settlement budgets. Their `hermes-api-settlement.json` rejects
reported-dollar claims, missing tokens, wrong model/biller and pending or
unpriced accounting. xAI, Google and routed Bedrock retain completion-only coverage.
Before creating the task, public account readback must identify the selected
connected personal account, its company and owner, and the authenticated caller.
Both run attribution fields must match that independently recorded owner.
@@ -1623,7 +1629,8 @@ pnpm test:e2e:runner -- --id hermes-bedrock-connections.runner-acpx-hermes-bedro
```
The explicit-only `hermes-native-interactions` suite declares question/reconnect
cells on local and Daytona, plus a separate local native Stop cell. One native Hermes
cells on local and Daytona, plus a separate local native Stop cell, for managed
OpenRouter, Claude API and OpenAI API profiles (nine explicit cells). One native Hermes
`clarify` call asks an ordered batch of single choice, multiple choice with a
custom answer, and free text. The browser reloads while the original run remains
active, checks the same durable request and complete form, then submits all
@@ -1634,8 +1641,9 @@ original succeeded run without a continuation. The final answer must contain
the reviewer's undisclosed free text returned by the native callback; a
semantic `request_human_input` call or a guessed completion cannot pass.
This suite uses the same managed OpenRouter account/model, public bounded
budgets, settled reported billing, pre-credential source admission and cleanup
This suite uses the same managed accounts/models, public bounded
budgets, settled reported OpenRouter or estimated direct-API billing,
pre-credential source admission and cleanup
pipeline. It allows one attempt and records its harness source digest. Its
browser-reconnect scope does not qualify controller restart, cancellation,
Stop, subscriptions, or Daytona until each corresponding live check passes.
@@ -1658,11 +1666,13 @@ mock control plane. Neither suite substitutes for the other.
### Hermes native image input (explicit-only)
`hermes-image-input` declares one local and one Daytona `image-code-complete`
cell. Its separate candidate profile uses `google/gemini-2.5-flash-lite` through
`hermes-image-input` declares local and Daytona `image-code-complete`
cells for OpenRouter, Claude API and OpenAI API (six explicit cells).
Its separate OpenRouter candidate profile uses `google/gemini-2.5-flash-lite` through
the selected managed OpenRouter account. The public model catalog reports image
input and tools; this metadata is preparation, not inference qualification.
The existing DeepSeek profile remains text-only.
The existing DeepSeek profile remains text-only. Direct API profiles use their
selected Claude/OpenAI models; successful inference is required to qualify image input.
The browser uploads a generated PNG in the ordinary task-creation flow. Its
eight-character hexadecimal code is present only in pixels, absent from the
@@ -1677,7 +1687,7 @@ can pass. It evaluates this gate after collecting account and billing evidence,
so wrong image bytes or forbidden tools cannot qualify on a correct final code.
The suite reuses managed account attribution, pre-credential source/runtime
admission, bounded company/agent budgets, per-run reported billing and cleanup.
admission, bounded company/agent budgets, per-run reported or estimated billing and cleanup.
It admits one attempt with no automatic retry. Remote selection still requires
the verified immutable image and authorized Daytona credential. Neither target
is qualified by the fixture's unit tests or public model discovery.
+3 -3
View File
@@ -146,10 +146,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
63, 12, 6, 8, 2, 2, 2, 40, 3, 2, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
63, 12, 6, 8, 2, 2, 2, 40, 9, 6, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
]);
expect(validateRunnerCatalog()).toHaveLength(725);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(725);
expect(validateRunnerCatalog()).toHaveLength(735);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(735);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+18 -11
View File
@@ -371,11 +371,18 @@ export const hermesApiConnectionProfiles: readonly RunnerProfileFixture[] = [
credential: choice.credential, model: choice.model,
modelQualification: {
source: "candidate_runner_profile",
qualificationId: `hermes:v2026.9.24:${choice.provider}:api:catalog-2026-10-07:pending`,
qualificationId: `hermes:v2026.9.24:${choice.provider}:api:catalog-${choice.provider === "openai" ? "2026-10-09" : "2026-10-07"}:pending`,
},
})),
];
/** API-billed Claude/OpenAI milestone; broader connection qualification stays pending. */
export const hermesBasicApiProfiles = hermesApiConnectionProfiles.filter(profile =>
profile.credential === "ANTHROPIC_API_KEY" || profile.credential === "OPENAI_API_KEY");
const hermesNativeInteractionProfiles = [
extendedHarnessProfiles.find(profile => profile.qualificationCandidate === "hermes")!, ...hermesBasicApiProfiles,
];
/** Region-bound Bedrock credentials enter only through a managed connection. */
export const hermesBedrockConnectionProfile = nativeProfile({
id: "runner-acpx-hermes-bedrock", label: "Hermes Bedrock (candidate)",
@@ -1216,14 +1223,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: HERMES_NATIVE_INTERACTION_SUITE, label: "Hermes native interactions", manualOnly: true,
description: "A native Hermes question batch receives exact answers through reconnect; browser Stop cancels an unanswered local callback and retires its owned process tree.",
groups: ["native"], profiles: [extendedHarnessProfiles.find(profile => profile.qualificationCandidate === "hermes")!],
groups: ["native"], profiles: hermesNativeInteractionProfiles,
environments: runnerEnvironments, tasks: [hermesNativeQuestionTask, hermesNativeQuestionStopTask],
excludedExecutionIds: [`${HERMES_NATIVE_INTERACTION_SUITE}.runner-acpx-hermes.daytona.native-question-batch-stop`], expectedMatrixSize: 3,
excludedExecutionIds: hermesNativeInteractionProfiles.map(profile => `${HERMES_NATIVE_INTERACTION_SUITE}.${profile.id}.daytona.native-question-batch-stop`), expectedMatrixSize: 9,
definitionMetadata: {
version: 8, qualification: "pending", scheduling: "explicit-only", accountMethod: "api_key", accountMode: "responsible_user",
version: 9, qualification: "pending", scheduling: "explicit-only", accountMethod: "api_key", accountMode: "responsible_user",
providerTurns: 1, lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", maximumAttemptsPerCell: 1,
objectiveAdmission: "production-delivery-guard-question-only",
budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, billing: "reported-cost-and-budget-health",
budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, billing: "reported-openrouter-or-estimated-direct-api-cost-and-budget-health",
coverage: "native-question-batch-browser-reconnect-exact-delivery-and-local-stop",
stopBoundary: "retained-unanswered-native-question-before-browser-click", stopProcessEvidence: "local-public-per-turn-owner-and-descendants-through-cleanup",
remoteStopQualification: "pending-separate-remote-retirement-observer", sourceDigest: hermesApiConnectionDefinitionDigest,
@@ -1232,16 +1239,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: HERMES_IMAGE_INPUT_SUITE, label: "Hermes native image input", manualOnly: true,
description: "An undisclosed image-only code passes through browser upload and the native Hermes prompt, with independent content and account checks.",
groups: ["native"], environments: runnerEnvironments, tasks: [hermesImageInputTask], expectedMatrixSize: 2,
groups: ["native"], environments: runnerEnvironments, tasks: [hermesImageInputTask], expectedMatrixSize: 6,
profiles: [nativeProfile({
id: "runner-acpx-hermes-vision", label: "Hermes vision (candidate)", provider: "acpx", acpxAgent: "hermes",
qualificationCandidate: "hermes", credential: "OPENROUTER_API_KEY", model: HERMES_IMAGE_INPUT_MODEL,
modelQualification: { source: "candidate_runner_profile", qualificationId: "hermes:v2026.9.24:openrouter:gemini-2.5-flash-lite:vision:catalog-2026-10-08:pending" },
})],
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", providerTurns: 1,
}), ...hermesBasicApiProfiles],
definitionMetadata: { version: 3, qualification: "pending", scheduling: "explicit-only", providerTurns: 1,
maximumAttemptsPerCell: 1, budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, lifecycle: "per-turn",
input: "browser-upload-authorized-native-image", oracle: "undisclosed-pixel-code-exact-bytes-no-file-tools",
billing: "per-run-reported-cost-and-budget-health", sourceDigest: hermesApiConnectionDefinitionDigest },
billing: "reported-openrouter-or-estimated-direct-api-cost-and-budget-health", sourceDigest: hermesApiConnectionDefinitionDigest },
},
{
id: "hermes-api-connections", label: "Hermes managed API connections", manualOnly: true,
@@ -1250,10 +1257,10 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
tasks: [{ ...openRouterBreadthTasks.find(task => task.id === "hello-complete")!, automaticRetryPolicy: "single_attempt" }],
expectedMatrixSize: 10,
definitionMetadata: {
version: 4, qualification: "pending", scheduling: "explicit-only", authenticatedDiscoveryDate: "2026-10-07",
version: 5, qualification: "pending", scheduling: "explicit-only", authenticatedDiscoveryDate: "2026-10-07", openaiDiscoveryDate: "2026-10-09",
accountMethod: "api_key", accountMode: "responsible_user", providerTurns: 1, expectedUserSource: "public-account-owner-before-task",
budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, maximumAttemptsPerCell: 1,
coverage: "api-account-native-completion-only", sourceDigest: hermesApiConnectionDefinitionDigest,
coverage: "api-account-native-completion-only", basicApiBilling: "complete-wire-tokens-rate-card-estimate-and-budget-health", sourceDigest: hermesApiConnectionDefinitionDigest,
},
},
{
+31 -10
View File
@@ -3,25 +3,46 @@ import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js";
import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
import { explicitlyRequestsFileOutput, explicitlyRequestsTaskDocumentOutput } from "../../server/src/services/native-runtime/native-deliverable-feedback.js";
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesOpenRouterWorkflow, isHermesConnectionSuite, HERMES_NATIVE_INTERACTION_SUITE, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, hasHermesNativeQuestionStop, hasHermesNativeQuestionStopCard, resolveHermesQualificationBudgetCents, HERMES_IMAGE_INPUT_SUITE, HERMES_IMAGE_INPUT_MODEL, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesApiSettlement, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesOpenRouterWorkflow, isHermesConnectionSuite, HERMES_NATIVE_INTERACTION_SUITE, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, hasHermesNativeQuestionStop, hasHermesNativeQuestionStopCard, resolveHermesQualificationBudgetCents, HERMES_IMAGE_INPUT_SUITE, HERMES_IMAGE_INPUT_MODEL, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
import { inflateSync } from "node:zlib";
describe("Hermes direct API billing oracle", () => {
it.each(["anthropic", "openai"] as const)("requires a scoped complete %s estimate and healthy post-run budgets", async biller => {
const model = biller === "anthropic" ? "claude-haiku-4-5-20251001" : "gpt-6-luna";
for (const fault of ["valid", "reported", "partial", "model", "biller", "provenance", "paused", "pending"]) {
const company = { id: "company", status: "active", budgetMonthlyCents: 200 };
const agent = { id: "agent", companyId: "company", status: fault === "paused" ? "paused" : "idle", pauseReason: null, budgetMonthlyCents: 200 };
const usage = { provider: biller, biller: fault === "biller" ? "unknown" : biller, model: fault === "model" ? "foreign" : model,
billingType: "metered_api", costStatus: fault === "reported" ? "reported" : "estimated", costUsd: 0.0042, costUsdExact: "0.004200000",
inputTokens: 40, outputTokens: 10, accountingReceiptReady: true, accountingUsageComplete: fault !== "partial",
pricingProvenance: { source: fault === "provenance" ? "agent_claim" : "rate_card",
version: biller === "anthropic" ? "anthropic-standard-2026-10-09" : "openai-standard-2026-09-30" } };
const run = { id: "run", companyId: "company", agentId: "agent", issueId: "task", status: "succeeded", usageJson: usage,
costAccountingPending: fault === "pending", costAccountedAt: "2026-10-09T14:00:00Z" };
const receipt = await captureHermesApiSettlement({ companyId: "company", agentId: "agent", issueId: "task", runId: "run", expectedBiller: biller, model,
api: { async get<T>(path: string) { return (path === "/api/companies/company" ? company : path === "/api/agents/agent" ? agent : run) as T; } } });
expect(receipt.checks.every(check => check.passed), `${biller}: ${fault}`).toBe(fault === "valid");
}
});
});
describe("Hermes native image input", () => {
const suite = runnerSuites.find(s => s.id === HERMES_IMAGE_INPUT_SUITE)!;
const cells = runnerMatrix.filter(e => e.suite.id === suite.id);
it("declares one bounded image journey per execution target on a separate vision model", () => {
expect(cells).toHaveLength(2);
it("declares bounded OpenRouter, Claude API and OpenAI API image journeys on both targets", () => {
expect(cells).toHaveLength(6);
expect(new Set(cells.map(c => c.environment.id))).toEqual(new Set(["local", "daytona"]));
expect(suite.manualOnly).toBe(true);
expect(isHermesConnectionSuite(suite.id)).toBe(true);
expect(cells.every(c => c.profile.model === HERMES_IMAGE_INPUT_MODEL && c.profile.credential === "OPENROUTER_API_KEY"
&& c.profile.qualificationCandidate === "hermes" && c.task.expectedRunCount === 1 && c.task.automaticRetryPolicy === "single_attempt")).toBe(true);
expect(new Set(cells.map(c => c.profile.credential))).toEqual(new Set(["OPENROUTER_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY"]));
expect(cells.every(c => c.profile.qualificationCandidate === "hermes" && c.task.expectedRunCount === 1 && c.task.automaticRetryPolicy === "single_attempt")).toBe(true);
expect(cells.filter(c => c.profile.credential === "OPENROUTER_API_KEY").every(c => c.profile.model === HERMES_IMAGE_INPUT_MODEL)).toBe(true);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === suite.id)).toBe(false);
const env = buildRunnerE2EProcessEnvironment({}, [cells[0]!]);
expect(JSON.parse(env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent: "hermes", model: HERMES_IMAGE_INPUT_MODEL }]);
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cells[0]!, suite: { ...suite, manualOnly: false } }])).toThrow("explicit");
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cells[0]!, profile: { ...cells[0]!.profile, qualificationCandidate: "pi" } }])).toThrow("explicit");
expect(suite.definitionMetadata).toMatchObject({ version: 2, qualification: "pending", providerTurns: 1,
expect(suite.definitionMetadata).toMatchObject({ version: 3, qualification: "pending", providerTurns: 1,
maximumAttemptsPerCell: 1, budgetMonthlyCents: 200 });
});
it("keeps the expected code out of prompt, filename and PNG metadata", () => {
@@ -116,15 +137,15 @@ describe("Hermes native browser questions", () => {
event("runtime_request.resolved", 3, { requestId: "request", turnId: "turn", action: "submit", response }),
];
const grade = (events: unknown[]) => hasExactHermesNativeQuestionResponse({ events, runId: "run", turnId: "turn", requestId: "request", questionSet, response });
it("declares three explicit bounded single-attempt cells with local-only Stop ownership proof", () => {
expect(cells).toHaveLength(3);
it("declares nine explicit bounded API cells with local-only Stop ownership proof", () => {
expect(cells).toHaveLength(9);
expect(suite.manualOnly).toBe(true);
expect(cells.filter(cell => cell.task.flow === "native_question_stop").map(cell => cell.environment.id)).toEqual(["local"]);
expect(cells.filter(cell => cell.task.flow === "native_question_stop").map(cell => cell.environment.id)).toEqual(["local", "local", "local"]);
expect(new Set(cells.map(cell => cell.environment.id))).toEqual(new Set(["local", "daytona"]));
expect(cells.every(cell => cell.profile.qualificationCandidate === "hermes" && ["native_question_completion", "native_question_stop"].includes(cell.task.flow)
&& cell.task.expectedRunCount === 1 && cell.task.automaticRetryPolicy === "single_attempt")).toBe(true);
expect(suite.definitionMetadata).toMatchObject({ qualification: "pending", providerTurns: 1, maximumAttemptsPerCell: 1, budgetMonthlyCents: 200,
lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", billing: "reported-cost-and-budget-health" });
lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", billing: "reported-openrouter-or-estimated-direct-api-cost-and-budget-health" });
expect(isHermesConnectionSuite(suite.id)).toBe(true);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(cell => cell.suite.id === suite.id)).toBe(false);
expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cells[0]!.id]))).toEqual([cells[0]]);
+23 -11
View File
@@ -451,12 +451,14 @@ export async function runHermesNativeQuestionStop(input: {
issueId: issue.id, connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider,
expectedResponsibleUserId: input.callerUserId, model: execution.profile.model, runs, expectedRunStatus: "cancelled" });
connectionChecks.forEach(value => check(value.id, value.passed, "Public native model and account attribution remain correct after Stop."));
let billing: Awaited<ReturnType<typeof captureHermesOpenRouterSettlement>> | undefined;
const expectedBiller = fixtures.aiConnection.binding.provider;
if (expectedBiller !== "openrouter" && expectedBiller !== "anthropic" && expectedBiller !== "openai") throw new Error("Unqualified Hermes Stop accounting provider");
let billing: Awaited<ReturnType<typeof captureHermesApiSettlement>> | undefined;
await pollUntil({ label: "cancelled Hermes run billing settlement", deadlineAt: Math.min(deadlineAt, Date.now() + 30_000), intervalMs: 200,
load: async () => billing = await captureHermesOpenRouterSettlement({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id,
issueId: issue.id, runId: final.run.id, expectedRunStatus: "cancelled" }), accept: receipt => receipt.checks.every(value => value.passed) });
billing!.checks.forEach(value => check(value.id, value.passed, "Cancelled native usage settles reported cost while the company and agent retain healthy budgets."));
await input.evidence("hermes-openrouter-settlement.json", billing);
load: async () => billing = await captureHermesApiSettlement({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id,
issueId: issue.id, runId: final.run.id, expectedRunStatus: "cancelled", expectedBiller, model: execution.profile.model }), accept: receipt => receipt.checks.every(value => value.passed) });
billing!.checks.forEach(value => check(value.id, value.passed, "Cancelled native usage settles with explicit reported or estimated provenance while budgets remain healthy."));
await input.evidence(expectedBiller === "openrouter" ? "hermes-openrouter-settlement.json" : "hermes-api-settlement.json", billing);
await input.capture("final-state", "Cancelled native question remains unanswerable", "final-state.png");
await input.evidence("api-state.json", { ...final, runs, run: final.run, checks, processes, nativeStop: identity });
complete = true;
@@ -500,9 +502,10 @@ export async function captureHermesApiBudgets(input: {
}
/** Observe settled billing and budget health before fixture cleanup pauses the agent. */
export async function captureHermesOpenRouterSettlement(input: {
export async function captureHermesApiSettlement(input: {
api: { get<T>(path: string): Promise<T> }; companyId: string; agentId: string; issueId: string; runId: string;
expectedRunStatus?: "succeeded" | "cancelled";
expectedBiller: "openrouter" | "anthropic" | "openai"; model?: string;
}) {
const [company, agent, run] = await Promise.all([
input.api.get<Record<string, unknown>>(`/api/companies/${input.companyId}`),
@@ -510,19 +513,23 @@ export async function captureHermesOpenRouterSettlement(input: {
input.api.get<Record<string, unknown>>(`/api/heartbeat-runs/${input.runId}`),
]);
const usage = record(run.usageJson), provenance = record(usage.pricingProvenance);
const direct = input.expectedBiller !== "openrouter";
const expectedVersion = input.expectedBiller === "anthropic" ? "anthropic-standard-2026-10-09"
: input.expectedBiller === "openai" ? "openai-standard-2026-09-30" : "hermes-openrouter-wire/v1";
const cost = usage.costUsd, exact = usage.costUsdExact;
return { observation: {
company: { id: company.id, status: company.status, pauseReason: company.pauseReason, budgetMonthlyCents: company.budgetMonthlyCents },
agent: { id: agent.id, companyId: agent.companyId, status: agent.status, pauseReason: agent.pauseReason, budgetMonthlyCents: agent.budgetMonthlyCents },
run: { id: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.issueId, status: run.status,
costAccountingPending: run.costAccountingPending, costAccountedAt: run.costAccountedAt,
usage: Object.fromEntries(["provider", "biller", "billingType", "costStatus", "costUsd", "costUsdExact", "inputTokens", "outputTokens", "accountingReceiptReady", "pricingProvenance"].map(key => [key, usage[key]])) },
usage: Object.fromEntries(["provider", "biller", "billingType", "model", "costStatus", "costUsd", "costUsdExact", "inputTokens", "outputTokens", "accountingReceiptReady", "accountingUsageComplete", "pricingProvenance"].map(key => [key, usage[key]])) },
}, checks: [
{ id: "billing-observation-scope", passed: company.id === input.companyId && agent.id === input.agentId && agent.companyId === input.companyId
&& run.id === input.runId && run.companyId === input.companyId && run.agentId === input.agentId && run.issueId === input.issueId },
{ id: "settled-openrouter-reported-cost", passed: run.status === (input.expectedRunStatus ?? "succeeded") && run.costAccountingPending === false && present(run.costAccountedAt)
&& usage.accountingReceiptReady === true && usage.biller === "openrouter" && usage.billingType === "metered_api" && usage.costStatus === "reported"
&& provenance.source === "provider_reported" && provenance.version === "hermes-openrouter-wire/v1"
{ id: direct ? `settled-${input.expectedBiller}-estimated-cost` : "settled-openrouter-reported-cost", passed: run.status === (input.expectedRunStatus ?? "succeeded") && run.costAccountingPending === false && present(run.costAccountedAt)
&& usage.accountingReceiptReady === true && usage.biller === input.expectedBiller && usage.billingType === "metered_api" && usage.costStatus === (direct ? "estimated" : "reported")
&& provenance.source === (direct ? "rate_card" : "provider_reported") && provenance.version === expectedVersion
&& (!direct || usage.provider === input.expectedBiller && usage.accountingUsageComplete === true && present(input.model) && usage.model === input.model)
&& typeof cost === "number" && Number.isFinite(cost) && cost >= 0 && typeof exact === "string" && /^(0|[1-9][0-9]{0,6})\.[0-9]{9}$/.test(exact) && Number(exact) === cost
&& typeof usage.inputTokens === "number" && Number.isSafeInteger(usage.inputTokens) && usage.inputTokens >= 0
&& typeof usage.outputTokens === "number" && Number.isSafeInteger(usage.outputTokens) && usage.outputTokens >= 0 && usage.inputTokens + usage.outputTokens > 0 },
@@ -531,6 +538,11 @@ export async function captureHermesOpenRouterSettlement(input: {
] };
}
/** Preserve the existing OpenRouter oracle's exact reported-price contract. */
export function captureHermesOpenRouterSettlement(input: Omit<Parameters<typeof captureHermesApiSettlement>[0], "expectedBiller" | "model">) {
return captureHermesApiSettlement({ ...input, expectedBiller: "openrouter" });
}
/** Grade public run/account/model metadata; a model's completion claim cannot supply it. */
export function gradeHermesApiConnection(input: {
companyId: string; agentId: string; issueId: string; connectionId: string; provider: string; model: string; expectedResponsibleUserId: string;
@@ -561,7 +573,7 @@ export function gradeHermesApiConnection(input: {
/** Explicit authenticated catalog choices; none is a production default or live qualification. */
export const hermesApiConnectionChoices = [
{ provider: "anthropic", credential: "ANTHROPIC_API_KEY", model: "claude-haiku-4-5-20251001" },
{ provider: "openai", credential: "OPENAI_API_KEY", model: "gpt-5.6-luna" },
{ provider: "openai", credential: "OPENAI_API_KEY", model: "gpt-6-luna" },
{ provider: "xai", credential: "XAI_API_KEY", model: "grok-4.7" },
// Google restricts 2.5 models to prior users even when they appear in the catalog.
// https://ai.google.dev/gemini-api/docs/deprecations
+8 -7
View File
@@ -1,5 +1,5 @@
import { runPlanTaskFlow } from "./plan-task-flow.js";
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite, isHermesOpenRouterWorkflow, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, runHermesNativeQuestionStop, HERMES_IMAGE_INPUT_SUITE, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesApiSettlement, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite, isHermesOpenRouterWorkflow, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, runHermesNativeQuestionStop, HERMES_IMAGE_INPUT_SUITE, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
import { isDeepStrictEqual } from "node:util";
import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
@@ -3055,20 +3055,21 @@ for (const execution of executions) {
{ id: "account-owner-preserved", passed: after.expectedResponsibleUserId === hermesApiAccountOwner.expectedResponsibleUserId });
await writeSanitizedJson(snapshotsDir, "hermes-api-owner-after-execution.json", after, secrets);
}
if (fixtures.aiConnection.binding.provider === "openrouter") {
const settlementBiller = fixtures.aiConnection.binding.provider;
if (settlementBiller === "openrouter" || !execution.profile.managedConnectionRouting && (settlementBiller === "anthropic" || settlementBiller === "openai")) {
const settlementScope = { companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue.id,
runId: selectedRuns[0]?.id ?? "missing" };
let receipt: Awaited<ReturnType<typeof captureHermesOpenRouterSettlement>> | undefined;
let receipt: Awaited<ReturnType<typeof captureHermesApiSettlement>> | undefined;
try {
await expect.poll(async () => {
receipt = await captureHermesOpenRouterSettlement({ api, ...settlementScope, expectedRunStatus });
receipt = await captureHermesApiSettlement({ api, ...settlementScope, expectedRunStatus, expectedBiller: settlementBiller, model: execution.profile.model });
return receipt.checks.every(check => check.passed);
}, { timeout: 30_000, message: "OpenRouter billing must settle without pausing the agent before cleanup" }).toBe(true);
}, { timeout: 30_000, message: "API billing must settle without pausing the agent before cleanup" }).toBe(true);
} finally {
if (receipt) {
await writeSanitizedJson(snapshotsDir, "hermes-openrouter-settlement.json", receipt, secrets);
await writeSanitizedJson(snapshotsDir, settlementBiller === "openrouter" ? "hermes-openrouter-settlement.json" : "hermes-api-settlement.json", receipt, secrets);
matcherResults.push(...receipt.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `hermesBilling.${check.id}`, expected: true },
passed: check.passed, detail: "Public reported cost and company/agent budget health must pass before teardown." })));
passed: check.passed, detail: "Public cost attribution, its reported/estimated provenance and company/agent budget health must pass before teardown." })));
}
}
}
@@ -143,6 +143,11 @@ describe("public repository paid workflow security", () => {
expect(manual).toContain('pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile');
expect(manual).toContain('PAPERCLIP_RUNNER_LOCK_SHA256=$lock_sha');
expect(manual).toContain('docker logout ghcr.io');
expect(manual).toContain('docker create --platform linux/amd64 --network none --entrypoint /bin/true "$immutable"');
expect(manual).toContain('docker cp "$container_id:/opt/paperclip-runner/provider-pack" "$runtime_dir/provider-pack"');
expect(manual).toContain('docker cp "$container_id:/usr/local/bin/paperclip-runnerd" "$runtime_dir/paperclip-runnerd"');
expect(manual).toContain('sha256sum paperclip-runnerd provider-pack/provider-pack.json');
expect(manual).toContain('sha256sum controller-runtime.tar.gz > controller-runtime.tar.gz.sha256');
});
it("uses the reviewed master branch for the first-party trusted PR workflow", async () => {